Optimizing Human-Robot Teaming Performance through Q-Learning-Based Task Load Adjustment and Physiological Data Analysis.

The transition to Industry 4.0 and 5.0 underscores the need for integrating humans into manufacturing processes, shifting the focus towards customization and personalization rather than traditional mass production. However, human performance during task execution may vary. To ensure high human-robot teaming (HRT) performance, it is crucial to predict performance without negatively affecting task execution. Therefore, to predict performance indirectly, significant factors affecting human performance, such as engagement and task load (i.e., amount of cognitive, physical, and/or sensory resources required to perform a particular task), must be considered. Hence, we propose a framework to predict and maximize the HRT performance. For the prediction of task performance during the development phase, our methodology employs features extracted from physiological data as inputs. The labels for these predictions-categorized as accurate performance or inaccurate performance due to high/low task load-are meticulously crafted using a combination of the NASA TLX questionnaire, records of human performance in quality control tasks, and the application of Q-Learning to derive task-specific weights for the task load indices. This structured approach enables the deployment of our model to exclusively rely on physiological data for predicting performance, thereby achieving an accuracy rate of 95.45% in forecasting HRT performance. To maintain optimized HRT performance, this study further introduces a method of dynamically adjusting the robot's speed in the case of low performance. This strategic adjustment is designed to effectively balance the task load, thereby enhancing the efficiency of human-robot collaboration.

Deciding when to cross in front of an autonomous vehicle: How child and adult pedestrians respond to eHMI timing and vehicle kinematics.

How autonomous vehicles (AVs) communicate their intentions to vulnerable road users (e.g., pedestrians) is a concern given the rapid growth and adoption of this technology. At present, little is known about how children respond to external Human Machine Interface (eHMI) signals from AVs. The current study examined how adults and children respond to the combination of explicit (eHMI signals) and implicit information (vehicle deceleration) to guide their road-crossing decisions. Children (8- to 12-year-olds) and adults made decisions about when to cross in front of a driverless car in an immersive virtual environment. The car sometimes stopped, either abruptly or gradually (manipulated within subjects), to allow participants to cross. When yielding, the car communicated its intent via a dome light that changed from red to green and varied in its timing onset (manipulated between subjects): early eHMI onset, late eHMI onset, or control (no eHMI). As expected, we found that both children and adults waited longer to enter the roadway when vehicles decelerated abruptly than gradually. However, adults responded to the early eHMI signal by crossing sooner when the cars decelerated either gradually or abruptly compared to the control condition. Children were heavily influenced by the late eHMI signal, crossing later when the eHMI signal appeared late and the vehicle decelerated either gradually or abruptly compared to the control condition. Unlike adults, children in the control condition behaved similarly to children in the early eHMI condition by crossing before the yielding vehicle came to a stop. Together, these findings suggest that early eHMI onset may lead to riskier behavior (initiating crossing well before a gradually decelerating vehicle comes to a stop), whereas late eHMI onset may lead to safer behavior (waiting for the eHMI signal to appear before initiating crossing). Without an eHMI signal, children show a concerning overreliance on gradual vehicle deceleration to judge yielding intent.

Frequency-encoded eye tracking smart contact lens for human-machine interaction.

Eye tracking techniques enable high-efficient, natural, and effortless human-machine interaction by detecting users' eye movements and decoding their attention and intentions. Here, a miniature, imperceptible, and biocompatible smart contact lens is proposed for in situ eye tracking and wireless eye-machine interaction. Employing the frequency encoding strategy, the chip-free and battery-free lens successes in detecting eye movement and closure. Using a time-sequential eye tracking algorithm, the lens has a great angular accuracy of <0.5°, which is even less than the vision range of central fovea. Multiple eye-machine interaction applications, such as eye-drawing, Gluttonous Snake game, web interaction, pan-tilt-zoom camera control, and robot vehicle control, are demonstrated on the eye movement model and in vivo rabbit. Furthermore, comprehensive biocompatibility tests are implemented, demonstrating low cytotoxicity and low eye irritation. Thus, the contact lens is expected to enrich approaches of eye tracking techniques and promote the development of human-machine interaction technology.

[Design and support performance evaluation of medical multi-position auxiliary support exoskeleton mechanism].

Aiming at the status of muscle and joint damage caused on surgeons keeping surgical posture for a long time, this paper designs a medical multi-position auxiliary support exoskeleton with multi-joint mechanism by analyzing the surgical postures and conducting conformational studies on different joints respectively. Then by establishing a human-machine static model, this study obtains the joint torque and joint force before and after the human body wears the exoskeleton, and calibrates the strength of the exoskeleton with finite element analysis software. The results show that the maximum stress of the exoskeleton is less than the material strength requirements, the overall deformation is small, and the structural strength of the exoskeleton meets the use requirements. Finally, in this study, subjects were selected to participate in the plantar pressure test and biomechanical simulation with the man-machine static model, and the results were analyzed in terms of plantar pressure, joint torque and joint force, muscle force and overall muscle metabolism to assess the exoskeleton support performance. The results show that the exoskeleton has better support for the whole body and can reduce the musculoskeletal burden. The exoskeleton mechanism in this study better matches the actual working needs of surgeons and provides a new paradigm for the design of medical support exoskeleton mechanism.

Operator selection for human-automation teaming: The role of manual task skill in predicting automation failure intervention.

Humans working in modern work systems are increasingly required to supervise task automation. We examined whether manual aircraft conflict detection skill predicted participants' ability to respond to conflict detection automation failures in simulated air traffic control. In a conflict discrimination task (to assess manual skill), participants determined whether pairs of aircraft were in conflict or not by judging their relative-arrival time at common intersection points. Then in a simulated air traffic control task, participants supervised automation which either partially or fully detected and resolved conflicts on their behalf. Automation supervision required participants to detect when automation may have failed and effectively intervene. When automation failed, participants who had better manual conflict detection skill were faster and more accurate to intervene. However, a substantial proportion of variance in failure intervention was not explained by manual conflict detection skill, potentially reflecting that future research should consider other cognitive skills underlying automation supervision.

Imposing Motion Variability for Ergonomic Human-Robot Collaboration.

OCCUPATIONAL APPLICATIONS"Overassistive" robots can adversely impact long-term human-robot collaboration in the workplace, leading to risks of worker complacency, reduced workforce skill sets, and diminished situational awareness. Ergonomics practitioners should thus be cautious about solely targeting widely adopted metrics for improving human-robot collaboration, such as user trust and comfort. By contrast, introducing variability and adaptation into a collaborative robot's behavior could prove vital in preventing the negative consequences of overreliance and overtrust in an autonomous partner. This work reported here explored how instilling variability into physical human-robot collaboration can have a measurably positive effect on ergonomics in a repetitive task. A review of principles related to this notion of "stimulating" robot behavior is also provided to further inform ergonomics practitioners of existing human-robot collaboration frameworks.

Background: Collaborative robots, or cobots, are becoming ubiquitous in occupational settings due to benefits that include improved worker safety and increased productivity. Existing research on human-robot collaboration in industry has made progress in enhancing workers' psychophysical states, by optimizing measures of ergonomics risk factors, such as human posture, comfort, and cognitive workload. However, short-term objectives for robotic assistance may conflict with the worker's long-term preferences, needs, and overall wellbeing.Purpose: To investigate the ergonomic advantages and disadvantages of employing a collaborative robotics framework that intentionally imposes variability in the robot's behavior to stimulate the human partner's psychophysical state.Methods: A review of "overassistance" within human-robot collaboration and methods of addressing this phenomenon via adaptive automation. In adaptive approaches, the robot assistance may even challenge the user to better achieve a long-term objective while partially conflicting with their short-term task goals. Common themes across these approaches were extracted to motivate and support the proposed idea of stimulating robot behavior in physical human-robot collaboration.Results: Experimental evidence to justify stimulating robot behavior is presented through a human-robot handover study. A robot handover policy that regularly injects variability into the object transfer location led to significantly larger dynamics in the torso rotations and center of mass of human receivers compared to an "overassistive" policy that constrains receiver motion. Crucially, the stimulating handover policy also generated improvements in widely used ergonomics risk indicators of human posture.Conclusions: Our findings underscore the potential ergonomic benefits of a cobot's actions imposing variability in a user's responsive behavior, rather than indirectly restricting human behavior by optimizing the immediate task objective. Therefore, a transition from cobot policies that optimize instantaneous measures of ergonomics to those that continuously engage users could hold promise for human-robot collaboration in occupational settings characterized by repeated interactions.

Teleoperator-Robot-Human Interaction in Manufacturing: Perspectives from Industry, Robot Manufacturers, and Researchers.

OCCUPATIONAL APPLICATIONSIndustrial robots have become an important aspect in modern industry. In the context of human-robot collaboration, enabling teleoperated robots to work in close proximity to local/onsite humans can provide new opportunities to improve human engagement in a distributed workplace. Interviews with industry stakeholders highlighted several potential benefits of such teleoperator-robot-human collaboration (tRHC), including the application of tRHC to tasks requiring both expertise and manual dexterity (e.g., maintenance and highly skilled tasks in sectors including construction, manufacturing, and healthcare), as well as opportunities to expand job accessibility for individuals with disabilities and older individuals. However, interviewees also indicated potential challenges of tRHC, particularly related to human perception (e.g., perceiving remote environments), safety, and trust. Given these challenges, and the current limited information on the practical value and implementation of tRHC, we propose several future research directions, with a focus on human factors and ergonomics, to help realize the potential benefits of tRHC.

Background The increasing prevalence of robots in industrial environments is attributed in part to advancements in collaborative robot technologies, enabling robots to work in close proximity to humans. Simultaneously, the rise of teleoperation, involving remote robot control, poses unique opportunities and challenges for human-robot collaboration (HRC) in diverse and distributed workspaces.Purpose There is not yet a comprehensive understanding of HRC in teleoperation, specifically focusing on collaborations involving the teleoperator, the robot, and the local or onsite workers in industrial settings, here referred to as teleoperator-robot-human collaboration (tRHC). We aimed to identify opportunities, challenges, and potential applications of tRHC through insights provided from industry stakeholders, thereby supporting effective future industrial implementations.Methods Thirteen stakeholders in robotics, specializing in different domains (i.e., safety, robot manufacturing, aerospace/automotive manufacturing, and supply chains), completed semi-structured interviews that focused on exploring diverse aspects relevant to tRHC. The interviews were then transcribed and thematic analysis was applied to group responses into broader categories, which were further compared across stakeholder industries.Results We identified three main categories and 13 themes from the interviews. These categories include Benefits, Concerns, and Technical Challenges. Interviewees highlighted accessibility, ergonomics, flexibility, safety, time & cost saving, and trust as benefits of tRHC. Concerns raised encompassed safety, standards, trust, and workplace optimization. Technical challenges consisted of critical issues such as communication time delays, the need for high dexterity in robot manipulators, the importance of establishing shared situational awareness among all agents, and the potential of augmented and virtual reality in providing immersive control interfaces.Conclusions Despite important challenges, tRHC could offer unique benefits, facilitating seamless collaboration among the teleoperator, teleoperated robot(s), and onsite workers across physical and geographic boundaries. To realize such benefits and address the challenges, we propose several research directions to further explore and develop tRHC capabilities.

A Literature Review on Safety Perception and Trust during Human-Robot Interaction with Autonomous Mobile Robots That Apply to Industrial Environments.

Occupational ApplicationsAutonomous mobile robots are used in manufacturing and warehousing industries, to transport material across the facility and deliver parts to work cells. Human workers might encounter or interact with these robots in aisle ways or at their workstation. It is important to consider factors that impact worker safety and trust when implementing autonomous mobile robots in the workplace. This paper reviews prior research that aims to improve the safety of human-robot interaction with autonomous mobile robots and identifies needs for future research. Researchers used a variety of questionnaires and behavioral assessment methods to measure perceived safety. Factors such as robot appearance, approach speed, and approach direction, significantly affect perceived safety. Additionally, projection of signals on the floor, turn signals, and haptic communication devices, can improve the predictability and overall safety of robot navigation.

Introduction: Autonomous mobile robots are rapidly emerging in the workplace, which potentially creates new hazards for human workers that interact with them.Purpose: We aimed to systematically review previous research on human-robot interaction with autonomous mobile robots that apply to industrial environments, and to identify research needs to improve worker safety and trust.Methods: We completed a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses methodology. We focused on articles that contained experiments with human participants and that included findings associated with improving safety and/or trust of workers who interact with mobile robots in industrial environments. We identified 50 articles that fit inclusion/exclusion criteria for the review.Results: Almost all of the reported experiments were conducted in a controlled laboratory setting. There were 27 different types of autonomous mobile robots. Only two studies involved industrial mobile robots that were commercially available and could be implemented in an industrial environment. Most studies used questionnaires, with the most common topic relating to participant perceptions of various robot traits, while few directly evaluated perceived safety and trust using questionnaires. Behavioral and physiological assessment methods were used in 70% and 8% of the studies, respectively. Separation distance between the participant and robot was the most common behavioral assessment method. A variety of robot characteristics were found to have a significant effect on human perception of safety and other similar concepts.Conclusions: Future research requires rigorous reporting of participant demographics and experience level with robots. We found that 34% and 44% of references failed to report the mean age of their participant sample and their experience with robots, respectively. Among several gaps that we identified in the literature were a lack of field experiments, sparse research involving multiple mobile robots, and limited use of industrial mobile robots in experiments with human participants.

A novel unsupervised dynamic feature domain adaptation strategy for cross-individual myoelectric gesture recognition.

Objective.Surface electromyography pattern recognition (sEMG-PR) is considered as a promising control method for human-machine interaction systems. However, the performance of a trained classifier would greatly degrade for novel users since sEMG signals are user-dependent and largely affected by a number of individual factors such as the quantity of subcutaneous fat and the skin impedance.Approach.To solve this issue, we proposed a novel unsupervised cross-individual motion recognition method that aligned sEMG features from different individuals by self-adaptive dimensional dynamic distribution adaptation (SD-DDA) in this study. In the method, both the distances of marginal and conditional distributions between source and target features were minimized through automatically selecting the optimal feature domain dimension by using a small amount of unlabeled target data.Main results.The effectiveness of the proposed method was tested on four different feature sets, and results showed that the average classification accuracy was improved by above 10% on our collected dataset with the best accuracy reached 90.4%. Compared to six kinds of classic transfer learning methods, the proposed method showed an outstanding performance with improvements of 3.2%-13.8%. Additionally, the proposed method achieved an approximate 9% improvement on a publicly available dataset.Significance.These results suggested that the proposed SD-DDA method is feasible for cross-individual motion intention recognition, which would provide help for the application of sEMG-PR based system.

Automation complacency on the road.

Given that automation complacency, a hitherto controversial concept, is already used to blame and punish human drivers in current accident investigations and courts, it is essential to map complacency research in driving automation and determine whether current research can support its legitimate usage in these practical fields. Here, we reviewed its status quo in the domain and conducted a thematic analysis. We then discussed five fundamental challenges that might undermine its scientific legitimation: conceptual confusion exists in whether it is an individual versus systems problem; uncertainties exist in current evidence of complacency; valid measures specific to complacency are lacking; short-term laboratory experiments cannot address the long-term nature of complacency and thus their findings may lack external validity; and no effective interventions directly target complacency prevention. The Human Factors/Ergonomics community has a responsibility to minimise its usage and defend human drivers who rely on automation that is far from perfect.Practitioner summary: Human drivers are accused of complacency and overreliance on driving automation in accident investigations and courts. Our review work shows that current academic research in the driving automation domain cannot support its legitimate usage in these practical fields. Its misuse will create a new form of consumer harms.

Simple manipulations of anthropomorphism fail to induce perceptions of humanness or improve trust in an automated agent.

Although automation is employed as an aid to human performance, operators often interact with automated decision aids inefficiently. The current study investigated whether anthropomorphic automation would engender higher trust and use, subsequently improving human-automation team performance. Participants performed a multi-element probabilistic signal detection task in which they diagnosed a hypothetical nuclear reactor as in a state of safety or danger. The task was completed unassisted and assisted by a 93%-reliable agent varying in anthropomorphism. Results gave no evidence that participants' perceptions of anthropomorphism differed between conditions. Further, anthropomorphic automation failed to bolster trust and automation-aided performance. Findings suggest that the benefits of anthropomorphism may be limited in some contexts.

Do concurrent task demands impact the benefit of automation transparency?

Automated decision aids typically improve decision-making, but incorrect advice risks automation misuse or disuse. We examined the novel question of whether increased automation transparency improves the accuracy of automation use under conditions with/without concurrent (non-automated assisted) task demands. Participants completed an uninhabited vehicle (UV) management task whereby they assigned the best UV to complete missions. Automation advised the best UV but was not always correct. Concurrent non-automated task demands decreased the accuracy of automation use, and increased decision time and perceived workload. With no concurrent task demands, increased transparency which provided more information on how the automation made decisions, improved the accuracy of automation use. With concurrent task demands, increased transparency led to higher trust ratings, faster decisions, and a bias towards agreeing with automation. These outcomes indicate increased reliance on highly transparent automation under conditions with concurrent task demands and have potential implications for human-automation teaming design.

Human-in-the-Loop Optimization of Wearable Robotic Devices to Improve Human-Robot Interaction: A Systematic Review.

This article presents a systematic review on wearable robotic devices that use human-in-the-loop optimization (HILO) strategies to improve human-robot interaction. A total of 46 HILO studies were identified and divided into upper and lower limb robotic devices. The main aspects from HILO were identified, reviewed, and classified in four areas: 1) human-machine systems; 2) optimization methods; 3) control strategies; and 4) experimental protocols. A variety of objective functions (physiological, biomechanical, and subjective), optimization strategies, and optimized control parameters configurations used in different control strategies are presented and analyzed. An overview of experimental protocols is provided, including metrics, tasks, and conditions tested. Moreover, the relevance given to training or adaptation periods was explored. We outline an HILO framework that includes current wearable robots, optimization strategies, objective functions, control strategies, and experimental protocols. We conclude by highlighting current research gaps and defining future directions to improve the development of advanced HILO strategies in upper and lower limb wearable robots.

Exploring user experience and performance of a tedious task through human-agent relationship.

Positive human-agent relationships can effectively improve human experience and performance in human-machine systems or environments. The characteristics of agents that enhance this relationship have garnered attention in human-agent or human-robot interactions. In this study, based on the rule of the persona effect, we study the effect of an agent's social cues on human-agent relationships and human performance. We constructed a tedious task in an immersive virtual environment, designing virtual partners with varying levels of human likeness and responsiveness. Human likeness encompassed appearance, sound, and behavior, while responsiveness referred to the way agents responded to humans. Based on the constructed environment, we present two studies to explore the effects of an agent's human likeness and responsiveness to agents on participants' performance and perception of human-agent relationships during the task. The results indicate that when participants work with an agent, its responsiveness attracts attention and induces positive feelings. Agents with responsiveness and appropriate social response strategies have a significant positive effect on human-agent relationships. These results shed some light on how to design virtual agents to improve user experience and performance in human-agent interactions.

Dialogue Management and Language Generation for a Robust Conversational Virtual Coach: Validation and User Study.

Designing human-machine interactive systems requires cooperation between different disciplines is required. In this work, we present a Dialogue Manager and a Language Generator that are the core modules of a Voice-based Spoken Dialogue System (SDS) capable of carrying out challenging, long and complex coaching conversations. We also develop an efficient integration procedure of the whole system that will act as an intelligent and robust Virtual Coach. The coaching task significantly differs from the classical applications of SDSs, resulting in a much higher degree of complexity and difficulty. The Virtual Coach has been successfully tested and validated in a user study with independent elderly, in three different countries with three different languages and cultures: Spain, France and Norway.

The frequency of positive and negative interactions influences relationship equity and trust in automation.

The purpose of this study was to 1) examine whether frequency of positive and negative interactions (manipulated via reliability) with a computer agent had an impact on an individual's trust resilience after a major error occurs and 2) empirically test the notion of relationship equity, which encompasses the total accumulation of positive and negative interactions and experiences between two actors, on user trust on a separate transfer task. Participants were randomized into one of four groups, differing in agent positivity and frequency of interaction, and completed both a pattern recognition task and transfer task with the aid of the same computer agent. Subjective trust ratings, performance data, compliance, and agreement were collected and analyzed. Results demonstrated that frequency of positive and negative interactions did have an impact on user trust and trust resilience after a major error. Additionally, it was shown that relationship equity has an impact on user trust and trust resilience. This is the first empirical demonstration of relationship equity's impact on user trust in an automated teammate.

Cognitive Analyses for Interface Design Using Dual N-Back Tasks for Mental Workload (MWL) Evaluation.

In the manufacturing environments of today, human-machine systems are constituted with complex and advanced technology, which demands workers' considerable mental workload. This work aims to design and evaluate a Graphical User Interface developed to induce mental workload based on Dual N-Back tasks for further analysis of human performance. This study's contribution lies in developing proper cognitive analyses of the graphical user interface, identifying human error when the Dual N-Back tasks are presented in an interface, and seeking better user-system interaction. Hierarchical task analysis and the Task Analysis Method for Error Identification were used for the cognitive analysis. Ten subjects participated voluntarily in the study, answering the NASA-TLX questionnaire at the end of the task. The NASA-TLX results determined the subjective participants' mental workload proving that the subjects were induced to different levels of mental workload (Low, Medium, and High) based on the ANOVA statistical results using the mean scores obtained and cognitive analysis identified redesign opportunities for graphical user interface improvement.

Single trial detection of error-related potentials in brain-machine interfaces: a survey and comparison of methods.

Objective.Error-related potential (ErrP) is a potential elicited in the brain when humans perceive an error. ErrPs have been researched in a variety of contexts, such as to increase the reliability of brain-computer interfaces (BCIs), increase the naturalness of human-machine interaction systems, teach systems, as well as study clinical conditions. Still, there is a significant challenge in detecting ErrP from a single trial, which may hamper its effective use. The literature presents ErrP detection accuracies quite variable across studies, which raises the question of whether this variability depends more on classification pipelines or on the quality of elicited ErrPs (mostly directly related to the underlying paradigms).Approach.With this purpose, 11 datasets have been used to compare several classification pipelines which were selected according to the studies that reported online performance above 75%. We also analyze the effects of different steps of the pipelines, such as resampling, window selection, augmentation, feature extraction, and classification.Main results.From our analysis, we have found that shrinkage-regularized linear discriminant analysis is the most robust method for classification, and for feature extraction, using Fisher criterion beamformer spatial features and overlapped window averages result in better classification performance. The overall experimental results suggest that classification accuracy is highly dependent on user tasks in BCI experiments and on signal quality (in terms of ErrP morphology, signal-to-noise ratio (SNR), and discrimination).Significance.This study contributes to the BCI research field by responding to the need for a guideline that can direct researchers in designing ErrP-based BCI tasks by accelerating the design steps.